Surgical system

ABSTRACT

A teleoperator system with telepresence is shown which includes right and left hand controllers ( 72 R and  72 L) for control of right and left manipulators ( 24 R and  24 L) through use of a servomechanism that includes computer ( 42 ). Cameras ( 46 R and  46 L) view workspace ( 30 ) from different angles for production of stereoscopic signal outputs at lines ( 48 R and  48 L). In response to the camera outputs a 3-dimensional top-to-bottom inverted image ( 30 I) is produced which, is reflected by mirror ( 66 ) toward the eyes of operator ( 18 ). A virtual image ( 30 V) is produced adjacent control arms ( 76 R and  76 L) which is viewed by operator ( 18 ) looking in the direction of the control arms. By locating the workspace image ( 30 V) adjacent the control arms ( 76 R and  76 L) the operator is provided with a sense that end effectors ( 40 R and  40 L) carried by manipulator arms ( 34 R and  34 L) and control arms ( 76 R and  76 L) are substantially integral. This sense of connection between the control arms ( 76 R and  76 L) and end effectors ( 40 R and  40 L) provide the operator with the sensation of directly controlling the end effectors by hand. By locating visual display ( 246 ) adjacent control arms ( 244 R and  244 L) image ( 240 I) of the workspace is directly viewable by the operator. (FIGS.  12  and  13 .) Use of the teleoperator system for surgical procedures also is disclosed. (FIGS.  7-9  and FIG.  13 .)

CROSS-REFERENCE TO RELATED APPLICATION

This is a continuation-in-part patent application which claims priorityfrom U.S. patent application No. 08/709,930 filed Sep. 9, 1996, which isa continuation of 07/823,932 filed Jan. 21, 1992 (ABN), the fulldisclosure of which is incorporated herein by reference.

FIELD OF THE INVENTION

This invention relates generally to teleoperator method and apparatus,and particularly to those which include means for providing the operatorof remote apparatus with the same sense as working directly with hishands at the worksite.

BACKGROUND OF THE INVENTION

Teleoperating, which is well known, includes the human performance oftasks at a remote location using manipulators. Telepresense includesproviding the teleoperator with the same feedback and control that hewould have were he actually at the worksite carrying out the operationwith his own hands. Telepresence operation generally includes use of astationary visual display, particularly a stereographic visual displayof the remote workspace. Stereoscopic television systems are well knownas shown, for example, in U.S. Pat. Nos. 4,562,463 and 4,583,117 and inU.K. Patent Application GB 2,040,134.

Remote manipulators employing stereoscopic TV viewing together withforce feedback also are well known as shown, for example, in an articleentitled, “Controlling Remote Manipulators Through KinestheticCoupling,” Bejczy et al, Computers in Mechanical Engineering, July 1983,pps. 48-60, and in an article entitled, “Stereo Advantage for aPeg-In-Hole Task Using a Force-Feedback Manipulator” by E. H. Spain,SPIE Vol. 1256 Stereoscopic Displays and Applications, 1990, pps.244-254. In the Bejczy et al. article, force-torque feedback isdisclosed. Also, in U.S. Pat. No. 3,921,445, a manipulator whichincludes force, torque and slip sensors of a type which may be employedwith the present invention is shown.

Even though the operator of prior art manipulators is provided with astationary three-dimensional image of the workspace, and manualcontrollers for control of the manipulators are provided with feedback,the operator is not provided with a sense of actually being present atthe worksite. The present invention is directed to a viewing arrangementfor use in a remote manipulation system which substantially adds to theoperator's sense of presence at the remote manipulator site.

BRIEF SUMMARY OF THE INVENTION

An object of this invention is the provision of an improved teleoperatorsystem and method which include an improved viewing system to enhancethe operator's sense of presence at remote manipulators controlled bythe operator from a remote location.

An object of this invention is the provision of an improved teleoperatorsystem and method of the above-mentioned type wherein an image ofmanipulator end effectors for viewing by the operator are sensed by theoperator as comprising an integral part of hand-controllers used by theoperator to control the end effectors, thereby giving the operator astrong sense of presence at the worksite.

An object of this invention is the provision of an improved teleoperatorsystem and method of the above-mentioned type which is well adapted foruse in a wide variety of applications including military, industrial,biomedical, and the like.

The present invention includes manipulators located at a worksite andwhich are controlled by hand-operated means at a remote operator controlstation. End effectors at the manipulators are used for manipulatingobjects located in a workspace at the worksite, and force-torquefeedback is employed for transmitting back to the operator mechanicalresistance encountered by the end effectors. Stereographic visualdisplay means provide the operator with an image of the workspace. Inaccordance with the present invention, the image is located adjacent thehand-operated means so that the operator looks in the direction of thehand-operated means for viewing the image adjacent the hand-operatedmeans. Either a real or virtual image of the workspace may be providedadjacent the hand-operated means. Display means for display of a realimage may be located adjacent the hand-operated means for direct viewingof the real image by the operator. For display of a virtual image of theworkspace, a mirror is located between the operator's eyes and thehand-operated means. In this case, display means provide a real imagewhich is inverted from top to bottom, which inverted image is viewed viathe mirror, which mirror inverts the image and provides the operatorwith a virtual image of the workspace, which appears to be locatedadjacent the hand-operated means. By locating the image of the workspaceadjacent the hand-operated means the operator is provided with a sensethat the end effectors and hand-operated means are substantiallyintegral despite the fact the end effectors are located at the worksiteand the hand-operated means are located at the remote operator'sstation. A stereophonic sound system may be included to provide theoperator with stereophonic sound from the worksite. Video camera meansare provided for viewing the workspace from which an image of theworkspace is obtained. Various other sensors and associated respondersmay be located at the worksite and operator's station, respectively, fortransmission of pressure, tactile, heat, vibration and similarinformation for enhanced telepresence operation.

Depending upon the application, different scaling may be provided in thetransmission of information between the operator's station and worksite.For example, for microassembly, microsurgery and like operationsinvolving small part manipulation, optical and/or video magnificationmay be employed to provide an enlarged 3-dimensional image for viewingby the operator. With similar scaling between the hand operated meansand manipulators, the perception of the operator is substantially thatwhich a miniature operator would have were he at the worksite.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention, together with other objects and advantages thereof, willbe better understood from the following description considered with theaccompanying drawings. It will be understood that the drawings are forpurposes of illustration and example only, and that the invention is notlimited thereto. In the drawings, wherein like reference charactersrefer to the same parts in the several views,

FIG. 1 is a diagrammatic showing of a teleoperator system embodying thepresent invention including side elevational views of a worksite andremote control operator's station;

FIG. 2 is an enlarged rear elevational view of the operator's stationtaken substantially along line 2—2 of FIG. 1;

FIG. 3 is an enlarged rear elevational view of the worksite takensubstantially along line 3—3 of FIG. 1;

FIG. 4 is a simplified side elevational view which is similar to FIG. 1,and showing dimensional relationships between elements at the worksiteand elements at the operator's station;

FIG. 5 is a diagrammatic view to illustrate visual perception by aminiature virtual eye, and FIG. 6 is a diagrammatic view to illustratevisual perception by the operator when image magnification is employed;

FIG. 7 is a diagrammatic view which is similar to that of FIG. 1 butshowing the teleoperator system used for telepresence surgery;

FIG. 8 is a rear elevational view of the operator's station shown inFIG. 7;

FIG. 9 is a rear elevational view of the worksite shown in FIG. 7;

FIGS. 10 and 11 are fragmentary side elevational views of modified formsof operator's station and manipulator, respectively, having increaseddegrees of freedom;

FIG. 12 is a side elevational view of a modified form of operator'sstation wherein display means are positioned for direct viewing by theoperator;

FIG. 13 is a rear elevational view of the modified form of operator'sstation shown in FIG. 12; and

FIG. 14 shows a fragmentary portion of the insertion portion of anendoscope for use with the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Reference now is made to FIGS. 1-3 wherein the teleoperator system isshown to include an operator's station 20 (FIGS. 1 and 2) and worksite22 (FIGS. 1 and 3). An operator 18 at the operator's station controlsmanipulator means 24 at the remote worksite. Manipulator means 24,comprising right and left manipulators 24R and 24L, respectively, areused for manipulating objects, such as object 26 which is shown locatedon a platform, or base, 28 within a workspace 30 shown in broken lines.For purposes of illustration only, and not by way of limitation, theright manipulator 24R is shown to comprise a housing 32R affixed to base28 and from which housing a telescopic arm 34R extends. The inner end34R1 of arm 34R is mounted for pivotal movement in any pivotal directionusing conventional mounting means. For example, the inner end of arm 34Rmay be mounted for pivotal movement about a horizontal pivot axis 36which pivot axis, in turn, is adapted for pivotal movement aboutvertical axis 38.

Arm 34R includes telescopic inner section 34R1 and outer section 34R2,which outer section is adapted both for axial movement into and out ofinner section 34R1 and for rotation about its longitudinal axis. An endeffector 40R is carried at the outer end of the arm which, for purposesof illustration, is shown to comprise a gripper. Motor means, not shown,control pivotal movement of arm 34R about pivot axes 36 and 38, axialand rotary movement of outer arm section 34R2 along and about thelongitudinal axis of the arm, and opening and closing of gripper 40R.The motor means, together with motor control circuits for control of themotors, may be included in housing 32R. The motors are under control ofa computer 42 connected thereto through right manipulator interface 44Rand the above-mentioned motor control circuits.

The left manipulator 24L is of substantially the same design as theright manipulator 24R and the same reference numerals, but with thesuffix L instead of R, are used to identify similar parts. For purposesof illustration, the left end effector 40L, shown in FIG. 3, is seen tocomprise cutting blades which operate to cut in the manner of a pair ofscissor blades.

The worksite is provided with a pair of video cameras 46R and 46L forviewing workspace 30 from different angles for production ofstereoscopic signal outputs therefrom at lines 48R and 48L. The angle γbetween the optical axes of the cameras shown in FIG. 3 is substantiallyequal to the operator's interocular viewing angle γ of an image of theworkspace as shown in FIG. 2.

The video camera outputs at lines 48R and 48L are supplied to an imagememory 50 for momentary storage of video fields of right and left imagesfrom the cameras. Fields of right and left images from image memory 50are alternately supplied through left/right switch means 52 to visualdisplay means 54, such as a television monitor, for alternate display ofthe two images at the face 54A of the monitor. Timing and control means56 provide timing and control signals to various elements of the system,including elements included in the stereographic display system, forsignal timing and control of the system. If digital storage means 50 areemployed, then conversion of the camera signal outputs to digital signalform by analog to digital converter means prior to storage, andconversion of the digital signal output from left/right switch means toanalog signal form in preparation for display at monitor 54 may beemployed.

An electrooptical device 58 at the face of the display means 54 controlspolarization of light received from display means 54 under control of aleft/right synchronizing signal from timing and control unit 56. Theleft and right image fields are viewed by operator 18 wearing a pair ofpassive polarized glasses 60 having right and left polarizing elements62 and 64 polarized in orthogonal directions. The polarization of lightfrom display 54 through electrooptical device 58 is synchronized fieldby field such that the right field is occluded from the left eye and theleft field is occluded from the right eye for stereographic viewing bythe operator. Other means for stereographic viewing of left and rightimage fields are well known, including, for example, those using activestereographic glasses, which may be used in the practice of thisinvention to provide the operator with a stereoscopic view of the remoteworkspace.

The vertical deflection coil connections for monitor 54 are reversed,causing the monitor to scan from bottom to top thereby creating atop-to-bottom inverted image 30I of workspace 30. Letters a, b, c, and dare used to identify corresponding corners of the workspace 30 andinverted workspace image 30I. The inverted workspace image 30I is viewedby the operator via a mirror 66 at the top of a table 68, which mirrorinverts image 30I to return the image as viewed by the operator to anupright position. Looking downwardly in the direction of the mirror, theoperator views a virtual image 30V of workspace 30. In accordance withone aspect of the present invention, the image viewed by the operator,which in the FIGS. 1-3 embodiment comprises a virtual image, is locatedadjacent controller means 70 used by the operator for control ofmanipulator means 24 at the worksite.

Controller means 70 are shown located beneath the table top 68 andinclude right and left controllers 72R and 72L for control of therespective right and left manipulators 24R and 24L. The right and leftcontrollers are of substantially the same design so that a descriptionof one applies to both. As with the manipulators, the suffixes R and Lare used to distinguish elements of the right controller from those ofthe left controller. For purposes of illustration, and not by way oflimitation, the right controller 72R is shown to comprise a housing 74Raffixed to the bottom of table top 68 and from which hand-operated means76R in the form of a telescopic control arm, or stick, extends.

The right and left control arms 76R and 76L are provided with the samedegrees of freedom as the associated manipulator arms 34R and 34L,respectively. For example, the inner end of control arm 76R is mountedfor pivotal movement about a horizontal pivot axis, corresponding tomanipulator pivot axis 36, which axis, in turn, is adapted for pivotalmovement about an intersecting vertical axis, corresponding tomanipulator axis 38. Control arm 76R also includes inner section 76R1and outer section 76R2, which outer section is adapted both for axialmovement into and out of inner section 76R1 and for rotation about itslongitudinal axis. It will be apparent that the control arm 76R isprovided with the same four degrees of freedom as the associatedmanipulator arm 34R. Additionally, sensor means 78R are located adjacentthe outer end of outer arm section 76R2 for use in controlling grippingaction of gripper 40R. Similar sensor means 78L adjacent the outer endof control arm 76L are adapted for use in controlling operation ofscissor blades 40L.

Right and left controllers 72R and 72L are included in a servomechanismsystem wherein mechanical motion of control arms 76R and 76L controlsthe position of manipulator arms 34R and 34L, and pressure on sensormeans 78R and 78L controls opening and closing of end effectors 40R and40L, respectively. In FIG. 1, right and left hand controller interfaces80R and 80L, respectively, are shown for connection of the controllersto computer 42. Servomechanisms for control of mechanical motion at aremote location are well known, including those which provide force andtorque feedback from the manipulator to the hand-operated controllermeans. Any suitable prior art servomechanism may be used in the practiceof the present invention, with those incorporating force and torquefeedback being particularly preferred for telepresence operation of thesystem. In the illustrated system, right and left microphones areincluded at the worksite, outputs from which microphones are amplifiedby right and left amplifiers and supplied to right and left speakers atthe operators' station for providing a stereophonic sound output toprovide the operator with an audio perspective present at the workspace.In FIG. 1, only the right channel of the stereophonic system is shownincluding right microphone 82R, right amplifier 86R, and right speaker88R. The left microphone and speaker are located directly behind therespective right microphone and speaker at the worksite and operator'scontrol station as viewed in FIG. 1. Obviously, earphones may beprovided for use by the operator in place of the speakers which wouldhelp to block out external noises at the operator's control station.Also, in FIG. 1 a light shield 54B at the monitor is shown for blockingdirect viewing of the monitor face by the operator.

Reference now is made to FIG. 4 wherein a simplified diagrammatic viewof the system illustrated in FIGS. 1-3 is shown and wherein variouslengths and angular positions are identified by reference characters. InFIG. 4, the optical path length between the cameras and a point F at theworkspace is identified by reference character L. A corresponding pathlength between the operator's eyes and point F at the virtual image ofthe workspace is identified by the distance a÷b, where a is the distancefrom the eyes of the operator to mirror 66, and b is the distance fromthe mirror to point F at the virtual image. Other dimensions showninclude the height G of the cameras above the pivot point of manipulatorarm 34R and corresponding height g of the operator's eyes above thepivot point of control arm 76R. With the control arm 76R at length d,the manipulator arm 34R adjusts to length D. Similarly, with the controlarm 76R at an angle β_(A), with the vertical, the manipulator arm 34R ispositioned at the same angle from vertical. The angle from vertical atwhich the cameras view the workspace and the eyes view the virtual imageof the workspace is identified by α.

Between elements of the worksite and operator station, the followingrelationships pertain:

 a+b=kL,  (1)

d=kD, and  (2)

g=kG  (3)

where k is a scale factor constant.

When k equals 1 such that a÷b=L, d=D and g=G, no scaling of worksitedimensions is required.

Any scale factor may be employed, the invention not being limited tofull-scale manipulation. For example, the worksite can be small,including microscopic in size, in which case the optical parameters,including distance to object, interocular distance and focal length, andmechanical and dimensional parameters are appropriately scaled.

By using appropriate scaling and image magnification and force andtorque feedback, and by locating the image 30V of the workspace 30adjacent hand-operated control means 76R and 76L, the operator isprovided with a strong sense of directly controlling the end effectors40R and 40L. The operator is provided with a sense that the endeffectors 40R and 40L and respective control arms 76R and 76L aresubstantially integral. This same sense of togetherness of thehand-operated control means and end effectors is not provided in priorart arrangements wherein the image viewed by the operator is not locatedadjacent the hand-operated control means. Even where the prior artincludes stereoscopic viewing and force and torque feedback, there is afeeling of disconnectedness of the hand notions from the visual imageobject being worked upon. The present invention overcomes this sense ofdisconnectedness by locating the workspace image where the operator'shands appear to exercise direct control over the end effectors.

For small-scale manipulation, such as required for surgicalapplications, it is desired to replicate the visual experience that aminiature observer would have were he closely adjacent the actualworksite. In FIG. 5, the virtual eye 90 of a hypothetical miniatureobserver is shown viewing an actual workspace. Light from a source at apoint X, Y, Z in the actual workspace produces a stimulus on theminiature observer's eye 90 at a point identified as X′/M. In FIG. 6, aneye 92 of an actual operator is shown viewing an enlarged image of thevirtual workspace produced by means of a video camera 94 used to viewthe actual workspace. The illustrated camera includes a light-receivinglens 96 and solid state imaging device such as a charge-coupled-device(CCD) array 98 where the point light source at X, Y, Z is shown imagedat point X_(i), Y_(i), Z_(i). With correct scaling, a correspondinglight source is produced at point MX_(i), MY_(i), MZ_(i) at either thereal or apparent position of the face of the visual display which, dueto stereoscopic operation of the system appears to the operator tooriginate from point MX, MY, MZ corresponding to point X, Y, Z at theactual workspace. At the retina of the actual eye 92, a stimulus isproduced at point X′ at proportionately the same position as point X′/Mat eye 90 of the hypothetical observer. This relationship is ensured byselecting a correctly scaled camera distance and lens focal length suchthat the optical magnification M_(o)=M/M_(v), where M is the desiredoverall magnification and M_(v) is the video magnification. A typicalvideo magnification M_(v), which equals the ratio of the CCD-array 98width to the display width, is about 40.

Reference now is made to FIGS. 7 through 9 wherein a modified form ofthis invention is shown for medical use. Here, right and leftmanipulators 100R and 100L are shown which are under control of rightand left controllers 102R and 102L, respectively. Elements of theimaging system are substantially the same as those employed in theimaging system illustrated in FIGS. 1-3 described above except that anenlarged virtual image 104V of actual workspace 104 is provided forviewing by the operator. Also, servomechanism elements for connection ofthe right and left controllers 102R and 102L to the respectivemanipulators 100R and 100L are substantially the same as those describedabove with reference to FIGS. 1-3. In the illustrated arrangement, theright and left manipulators are of substantially the same constructionas are the right and left controllers, such that a description of onemanipulator and one controller applies to both. Again, suffixes R and Lare used to distinguish between right and left elements thereof.

The manipulators include outer control sections 100RA and 100LA andinsertion sections 100RB and 100LB, which insertion sections are adaptedfor insertion into a body cavity through cylindrical tubes, or cannulas,not shown. For purposes of illustration, the manipulators are showninserted through the abdomen wall 106 of a subject. As is wellunderstood, for laparoscopic surgical procedures, wall 106 is separatedfrom internal organs by insufflation wherein a gas is introduced intothe abdomen by any suitable means not shown. Manipulator motors andassociated motor control circuits are contained in the outer controlsections 100RA and 100LA of the manipulators for control of theinsertion section. The manipulators, together with a laparoscope 108 forviewing organs within the cavity, are carried by a fixed rail 110forming part of a surgical table upon which the subject is supported.

The insertion sections 100RB and 100LB of the manipulators may be ofsubstantially the same design as manipulator arms 34R and 34L describedabove with reference to the FIGS. 1-3 embodiment. The insertion sectionsare of relatively small size for use inside the body. Insertion section100RB includes telescopic inner section 112R1 and outer section 112R2,which outer section is adapted for both axial movement into and out ofinner section 112R1 and for rotation about its longitudinal axis. Endeffectors 114R and 114L are carried at the outer ends of the respectiveright and left sections 112R2 and 112L2 for manipulation of organ 116.The inner section 112R1 is adapted for pivotal movement aboutintersecting perpendicular axes located substantially at point P wherethe insertion section intersects wall 106. Exclusive of operation of endeffectors 114R and 114L the manipulator arms each are provided with fourdegrees of freedom, the same as in the embodiment shown in FIGS. 1-3.End effectors 114R and 114L simply may comprise, essentially,microsurgical instruments with their handles removed including, forexample, retractors, electrosurgical cutters and coagulators,micro-forceps, microneedle holders, dissecting scissors, blades,irrigators, and sutures.

Laparoscope 108 for viewing the workspace 104 is shown comprising anouter operating section 108A and insertion section 108B. The outer endsection 120 of insertion section 108B is axially and rotatably movablewithin the inner end 122 thereof, and is provided with a pair of imagetransmission windows 124, 124 for stereoscopic viewing of workspace 104.The laparoscope also is provided with illuminating means, not shown, forilluminating the workspace, and with liquid inlet and outlet means, notshown, for flow of liquid past the windows. Video camera means withinsection 108A are responsive to light received through the viewingwindows for generation of left and right electronic images at outputlines 48R and 48L for connection to image memory 50. A magnified3-dimensional image 104I is produced at display means 54 for viewing bythe operator wearing cross-polarized classes 60 via mirror 66. As withthe embodiment shown in FIGS. 1-3, a virtual image 104V of the workspace104 is produced adjacent control arms 130R and 130L of controllers 102Rand 102L. Control arms 130R and 130L are of the same type as controlarms 76R and 76L included in the FIGS. 1-3 embodiment described above.They include telescopic inner and outer sections 132R1 and 132R, and132L1 and 132L2. Sensor means 134R and 134L located adjacent the outerends of the control arms control operation of end effectors 114R and114L, respectively, in the manner described above with reference toFIGS. 1-3. It here will be noted that the angle from vertical at whichthe image is viewed by the operator need not equal the angle fromvertical at which the object is viewed by the cameras. In thearrangement illustrated in FIGS. 7-9, the operator is shown to view theimage 104V at an angle θ from vertical (FIG. 7) whereas the object 116is shown as viewed directly downwardly. With no external reference, thesense of vertical within a body is not particularly great, and noconfusion is produced in the mind of the operator as a result of thedifferent observer and camera viewing angles relative to vertical.

With the FIGS. 7-9 embodiment, not only is a magnified virtual image104V of the workspace provided for viewing by the operator, but controlarms 130R and 130L of greater length than the length of the manipulatorinsertion sections 100RB and 100LB are employed. Servomechanism scalingof axial movement of the telescopic control arms is provided such thataxial extension or retraction thereof results in a smaller extension orretraction of the telescopic insertion sections. Angular pivotal motionof the control arms 130R and 130L produces the same angular pivotalmotion of insertion sections 100RB and 100LB, and rotational movement ofthe end sections 132R2 and 132L2 of the control arms produces the samerotational motion of end sections 112R2 and 112L2 of the insertionsections of the right and left manipulators, without scaling. Thisembodiment of the invention, with its magnified image, is of particularuse in the area of microsurgery, and especially in those cases where thesurgeon cannot reach an area by hand because of size constraints.

The present invention is not limited to use with manipulators having anyparticular number of degrees of freedom. Manipulators with differentdegrees of freedom which are well known in the art may be used in thepractice of this invention. In FIGS. 10 and 11, to which reference nowis made a controller 140 and manipulator 142, respectively, are shownwhich include a wrist joint to provide the same with additional freedomof movement. The illustrated controller 140 includes a housing 144affixed to the bottom of table top 68 upon which table mirror 66 islocated. An enlarged virtual image 146V of actual workspace 146 isprovided adjacent the operator's hand 148 viewable by the operator whenlooking downwardly onto the mirror 66 in a manner described above.

A control arm 150L comprising inner and outer sections 150L1 and 150L2,respectively, is mounted within housing l44 for pivotal movement in anypivotal direction as indicated by intersecting double-headed arrows 152and 154. The outer section 150L2 is adapted for axial movement into andout of inner section 150L1 in the direction of double-headed arrow 156.It also is adapted for rotation about its longitudinal axis in thedirection of double-headed arrow 158. In this embodiment, the controlarm includes an end section 160 pivotally attached to outer section150L2 by wrist joint 162 for pivotal movement in the direction ofdouble-headed arrow 164. End section 160 comprises axially aligned innerand outer sections 160A and 160B, the outer section 160B of which isrotatable about its longitudinal axis in the direction of double-headedarrow 166. As with the above-described arrangements, sensor means 168are located adjacent the free end of the control arm for operation of anend effector 170 at manipulator 142 shown in FIG. 11.

Referring to FIG. 11, end effector 170 is shown to comprise a pair ofmovable jaws attached to a wrist 172 comprising axially aligned links172A and 172B. Outer link 172B is rotatable about its longitudinal axisrelative to inner link 172A by motor means, not shown, in the directionof double-headed arrow 166M in response to rotation of section 160B ofthe hand-operated control unit in the direction of arrow 166. Wrist link172A is pivotally attached to manipulator forearm 174 for pivotalmovement in the direction of double-headed arrow 164M in response topivotal movement of end section 160 of the hand-operated control meansabout pivot axis 162. Forearm 174 is longitudinally axially movable inthe direction of double-headed arrow 156M in response to axial movementof outer section 150L2 of control arm 150L in the direction ofdouble-headed arrow 156. It also is rotatable about its longitudinalaxis in the direction of double-headed arrow 158M in response torotation of outer section 150L2 of control arm 150L in the direction ofdouble-headed arrow 158. Additionally, it is pivotally movable aboutpoint 176 in the directions of double-headed arrows 152M and 154M inresponse to pivotal movement of control arm 150L in the directions ofdouble-headed arrows 152 and 154, respectively. For biomedical use, suchas remote laparoscopic surgery, pivot point 176 is substantially locatedat the level of abdominal wall 178 through which the manipulatorextends. In FIG. 11, manipulator arm 174 is shown extending through acannula 180 which penetrates the abdominal wall.

The outer operating end of the manipulator is adapted for attachment toa supporting rail, not shown, of the surgery table upon which thesubject is supported. It includes an end effector drive motor 182 foropening and closing of gripper 170. Wrist drive motor 184 controlspivotal movement of wrist 172 in the direction of double-headed arrow164M, and extension drive motor 186 controls axial movement ofmanipulator arm 174 in the direction of double-headed arrow 156M.Forearm pivotal control motors and linkages, identified generally byreference numeral 188, provide for pivotal movement of arm 174 aboutpivot point 176 in the directions of arrows 152M and 154M. Pivotalmotion about point 176 is provided by simultaneous lateral movement ofthe outer operating end of the manipulator and pivotal movement of arm174. Movements are coordinated such that the center of rotation offorearm 174 is fixed in space at point 176 at the level of the abdominalwall.

Controller 140 and manipulator 142 are included in a system such asshown in FIGS. 7, 8 and 9 which includes a second controller andmanipulator for use by the operator's right hand, and associatedservomechanism means of any suitable type, not shown, for remote controlof the manipulators by the hand-operated controllers. Video camera meansat the worksite, such as shown in FIG. 9, together with display means,such as shown in FIG. 7, are employed for providing the operator with animage of the workspace at a location adjacent the left and righthand-operated control means. By using manipulators with a wrist joint,an added degree of freedom is provided for increased maneuverability andusefulness thereof. However, as noted above, the present invention isnot limited to use with manipulators with any particular degree offreedom.

Reference now is made to FIGS. 12 and 13 wherein a modified form of thisinvention is shown which provides for direct viewing of a 3-dimensionalimage 240I of a workspace, not shown. In FIGS. 12 and 13, only theoperator's station is shown, which includes right and left controllers242R and 242L and associated right and left hand-operated means 244R and244L which may be of the same type as controllers and control armsdescribed above. The operator's station is adapted for remote control ofmanipulators which also may be of the above-described type. The3-dimensional image 240I of the workspace is provided by visual displaymeans 246 in conjunction with electrooptical device 58 at the face ofthe display means and cross-polarized glasses 60 worn by the operator,to which display means left and right video fields from left and rightvideo cameras that view the workspace are alternately supplied, all inthe manner described in detail above. End effector and object images 248and 250, respectively, are shown within the workspace image as viewed byvideo cameras at the worksite. The display means 246 is located adjacentthe left and right hand-operated means 244R and 244L for direct viewingby the operator. With this arrangement, the end effector and objectimages together with the hand-operated means 244R and 244L aresimultaneously viewable by the operator. Since the hand-operated meansalso are visible, the operator is provided with a visual sense ofconnection between the end effector means and hand-operated meanswhereby they appear substantially as being integral.

Reference now is made to FIG. 14 wherein the distal end portion, or tip,260 of the insertion section of an endoscope is shown which is ofsubstantially the same type as shown in the above-mentioned publicationentitled “Introduction to a New Project for National Research andDevelopment Program (Large-Scale Project) in FY 1991,” which endoscopemay be used in the practice of the present invention. The insertion endof the endoscope includes a pair of spaced viewing windows 262R and 262Land an illumination source 264 for viewing and illuminating a workspaceto be observed. Light received at the windows is focused by objectivelens means, not shown, and transmitted through fiber-optic bundles to apair of cameras at the operating end of the endoscope, not shown. Thecamera outputs are converted to a 3-dimensional image of the workspacewhich image is located adjacent hand-operated means at the operator'sstation, not shown. Right and left steerable catheters 268R and 268Lpass through accessory channels in the endoscope body, which cathetersare adapted for extension from the distal end portion, as illustrated.End effectors 270R and 270L are provided at the ends of the catheterswhich may comprise conventional endoscopic instruments. Force sensors,rot shown, also are inserted through the endoscope channels. Steerablecatheters which include control wires for controlling bending of thecatheters and operation of an end effector suitable for use with thisinvention are well known. Control motors for operation of the controlwires are provided at the operating end of the endoscope, which motorsare included in a servomechanism of a type described above for operationof the steerable catheters and associated end effectors from a remoteoperator's station. As with the other embodiments, the interfacingcomputer in the servomechanism system remaps the operator's hand motioninto the coordinate system of the end effectors, and images of the endeffectors are viewable adjacent the hand-operated controllers in amanner described above. With this embodiment, the operator has thesensation of reaching through the endoscope to put his hands directly onthe end effectors for control thereof. Endoscopes of different types maybe employed in this embodiment of the invention so long as they includeone or more accessory channels for use in control of end effector means,and suitable viewing means for use in providing a visual display of theworkspace. For example, gastric, colonscopic, and like type, endoscopesmay be employed.

The invention having been described in detail in accordance withrequirements of the Patent Statutes, various other charges andmodifications will suggest themselves to those skilled in this art. Forexample, as noted above, the invention may include the use of tactilefeedback to provide the subtle sensations for palpation and formanipulating tissues and instruments. To provide this feedback, tactilesensor arrays may be included on the end effectors which are coupled totactile sensor stimulator arrays on the hand-operated control means,which reproduce the tactile sensation on the operator's hands. A varietyof transduction technologies for teleoperator tactile sensing are knownincluding resistive/conductive, semiconductor, piezoelectric capacitiveand photoelectric. Hand-operated control means and manipulators ofdifferent types may be employed using a wide variety of well-knownmechanisms and electromechanical elements including, for example,gimbals, linkages, pulleys, cables, drive belts and bands, gears,optical or electromagnetic position encoders, and angular and linearmotors. Force feedback to the operator requires use of body contact withhand-operated control means. Both hand grip type hand controllers suchas those illustrated, and control brace type hand controllers are welladapted for use with the present invention for force feedback to theoperator. Control brace hand controllers include use of structures withpositive sensors mounted on the operator at joints for measuring jointangles. Force feedback then can be applied to each joint. Similarly,light fabric gloves with variable-resistance or fiber-optic flex sensorsmounted on the joints for measuring bending of individual fingers may beused. Gloves of this type also may be provided with force feedback toprovide for telepresence interaction with real objects. Regardless ofthe type of hand-operated control means employed, an image of theworkspace is produced adjacent thereto to provide the operator with asense that the end effector means and hand-operated control means aresubstantially integral. Also, as noted above, servomechanisms of manydifferent types are well known in the robotic and teleoperator systemarts, and the invention is not limited to any particular type. Thosethat include force and torque feedback to the operator are preferred tocontribute to a telepresence sense of operation. In addition, manydifferent means for producing a stereoscopic image of the workspace areknown. For example, instead of using two cameras, a single camera may beemployed together with switched cross-polarizing elements in the imagereceiving path. In this case, a pair of spaced stereoscopic lenses areused for viewing the workspace from different angles and providing firstand second images thereof to the camera. In the FIG. 9 arrangement,wherein a laparoscope is shown, other types of endoscopes may be usedfor viewing the workspace. As noted above, the invention is not limitedto any particular application or use. In the biomedical field, usesinclude, for example, open surgery, including surgery from a remotelocation, microsurgery, and minimum invasive surgery such aslaparoscopic and endoscopic surgery. Laboratory use includingmicroscopic manipulation also is contemplated. Industrial use of theinvention include, for example, hazardous materials handling, remoteoperations, microassembly, and the like. Military and undersea use ofthe teleoperator system of this system are apparent. It is intended thatthe above and other such changes and modifications shall fall within thespirit and scope of the invention defined in the appended claims.

What is claimed is:
 1. An endoscopic surgical system comprising: acontrol section and an insertion section, wherein the insertion sectionis insertable into a patient through an aperture in a body wall to alocation adjacent a surgical worksite in the patient; the insertionsection comprising an elongate forearm portion having proximal anddistal ends and a longitudinal centerline extending between the proximaland distal ends, and an end effector, the end effector coupled to theforearm portion distal end in such a manner as to provide the endeffector with at least two degrees of freedom of movement within thepatient; the control section comprising an operator control stationhaving a manual controller, and a drive system having plurality ofcontrol motors and linkages, the drive system operatively coupled toboth the insertion section and the control station so that an operatoris able to manually move the controller to operate the insertion sectionin such a manner as at least: to move the forearm portion in at leastone degree of freedom relative to a rotational center, wherein therotational center is generally adjacent to the body wall aperture, andthe rotational center is generally adjacent the centerline of to forearmportion, to rotate the forearm portion about the proximal longitudinalcenterline of the forearm portion, to move the end effector in at leasttwo degrees of freedom of movement within the patient relative to theforearm portion within the patient's body distal of the body wall duringmovement, and to manipulate tissue at the surgical site with the endeffector.
 2. The surgical system of claim 1, wherein the drive system isfurther operatively coupled to the insertion section and the controlstation so that an operator is able to manually move the controller tooperate the insertion section to rotate the end effector generallyaxially relative to the distal longitudinal centerline of the forearmportion within the patient's body distal of the body wall.
 3. Thesurgical system of claim 1, wherein the coupling of the drive system tothe insertion section restrains the motion of the forearm portion sothat the longitudinal centerline of the forearm like remainssubstantially aligned to pass through the rotational center.
 4. Thesurgical system of claim 1, wherein the drive system is furtheroperatively coupled to the insertion section and the control station sothat an operator is able to manually move the controller to operate theinsertion section to move the end effector generally transversely in atleast one degree of freedom relative to the distal longitudinalcenterline of the forearm portion within the patients body distal of thebody wall.
 5. The surgical system of claim 4, wherein to end effector iscouplable to the forearm portion by connection to a pivotal wrist membercoupled to the forearm portion adjacent the distal end of the forearmportion, the wrist member actuatable to pivot in at least one degree offreedom of relative to the forearm portion.
 6. The surgical system ofclaim 4, wherein the end effector is couplable to the forearm portion byconnection to a bendable member coupled to the forearm portion.
 7. Thesurgical system of claim 6, wherein the forearm portion includes atleast one elongate steering control element coupled to the drive systemand the bendable member, the control element actuatable to steer thebendable member in at least one degree of freedom of relative to theforearm portion.
 8. The surgical system of claim 7, wherein the at leastone elongate steering control element comprises a plurality of cables.9. The surgical system of claim 1, wherein the distal longitudinalcenterline of the forearm portion is generally aligned collinearly tothe proximal longitudinal centerline of the forearm portion, the forearmportion defining a generally straight body.
 10. An endoscopic surgicalsystem having a control section operatively coupled to a surgicalmanipulator assembly; the control section comprising an operator controlstation having a manual controller; and the surgical manipulatorassembly comprising a drive system and an insertion section insertableinto a patient through an aperture in a body wall to a location adjacenta surgical worksite in the patient; the insertion section including anend effector and an elongate forearm assembly having proximal end, adistal end and a longitudinal centerline extending between the proximaland distal ends; the end effector being couplable to the forearmassembly in such a manner that the end effector is disposed adjacent theforearm assembly distal end; the drive system having plurality ofactuator motors and linkages, the drive system operatively coupled toboth the insertion section and the control station so that an operatoris able to manually move the controller to operate the insertion sectionin such a manner as at least: to move the forearm assembly in at leastone degree of freedom about an rotational center, wherein the rotationalcenter is generally adjacent to, the body wall aperture, and therotational center is generally adjacent the centerline of the proximalend of the forearm assembly; to rotate the forearm assembly about theproximal longitudinal centerline; and to provide the end effector withat least one motion actuation to manipulate tissue at the surgical site.11. The surgical system of claim 10, wherein the drive system is furtheroperatively coupled to the insertion section and to control station sothat an operator is able to manually move the controller to operate theinsertion section to rotate the end effector generally axially relativeto the distal longitudinal centerline of the forearm assembly within thepatient's body distal of to body wall.
 12. The surgical system of claim10, wherein the coupling of the drive system to the insertion sectionrestrains the motion of the forearm assembly so that the longitudinalcenterline of the forearm like remains substantially aligned to passthrough the rotational center.
 13. The surgical system of claim 10,wherein the drive system is further operatively coupled to the insertionsection and the control station so that an operator is able to manuallymove the controller to operate the insertion section to move the endeffector generally transversely in at least one degree of freedomrelative to the distal longitudinal centerline of the forearm assemblywithin the patient's body distal of the body wall.
 14. The surgicalsystem of claim 13, wherein to end effector is couplable to the forearmassembly by connection to a pivotal wrist member coupled to the forearmassembly adjacent the distal end of the forearm assembly, the wristmember actuatable to pivot in at least one degree of freedom of relativeto the forearm assembly.
 15. The surgical system of claim 13, whereinthe end effector is couplable to the forearm assembly by connection to abendable member coupled to the forearm assembly.
 16. The surgical systemof claim 15, wherein the forearm assembly includes at least one elongatesteering control element coupled to the drive system and the bendablemember, the control element actuatable to steer the bendable member inat least one degree of freedom of relative to the forearm assembly. 17.The surgical system of claim 16, wherein the at least one elongatesteering control element coupled comprises a plurality of cables. 18.The surgical system of claim 15, wherein the forearm assembly includes alongitudinal channel, the bendable member and end effector beingremovably insertable through the longitudinal channel so that the endeffector extends distally from the forearm assembly to a point adjacentthe surgical site.
 19. The surgical system of claim 10, wherein thedistal longitudinal centerline of the forearm assembly is generallyaligned collinearly to the proximal longitudinal centerline of theforearm assembly, the forearm assembly defining a generally straightbody.
 20. A robotic medical instrument system having a control sectionoperatively coupled to an instrument manipulator assembly; the controlsection comprising an operator control station having a operator inputdevice; and the instrument manipulator assembly comprising a drivesystem, and an insertion section at least a portion of which isinsertable through an aperture in a patient to a location adjacent aworksite within the patient's body, the insertion section including: abody member having a proximal end, a distal end and a longitudinalcenterline extending between the proximal and distal ends; at least onesteerable bendable member having a proximal end and a distal end, thesteerable bendable member coupled to the body member and arranged toextend distally of the body member; and an end effector coupled adjacentthe distal end of the steerable bendable member; the drive systemincluding a plurality of actuator motors and linkages, the drive systemoperatively coupled to both the insertion section and the controlstation so that an operator is able use the input device to operate theinsertion section in such a manner as at least to move the steerablebendable member so as to provide the end effector with at least onedegree of freedom of motion relative to the body member.
 21. Thesurgical system of claim 20, wherein the end effector is movablyactuatable, and wherein the drive system is further operatively coupledto both the insertion section and the control station so that anoperator is able use the input device to operate the insertion sectionin such a manner as to provide the end effector with at least one motionactuation to manipulate tissue at the surgical site.
 22. The surgicalsystem of claim 20, wherein: the aperture in the patient comprises aminimally invasive incision; at least a portion of the body member isinsertable through the incision; and the drive system is furtheroperatively coupled to both the insertion section and the controlstation so that an operator is able use the input device to operate theinsertion section in such a manner as to move the body member in atleast one degree of freedom with respect to an insertion point adjacentto the incision, the motion of the body member coordinated so that thelongitudinal centerline of the body member remains aligned tosubstantially pass through the insertion point.
 23. The surgical systemof claim 22, wherein the at least one degree of freedom of motion of thebody member with respect to the insertion point includes one of: (a)pivotal rotation of the body member in a first direction about theinsertion point; (b) pivotal rotation of the body member in a seconddirection about the insertion point, perpendicular to the firstdirection; (c) translation of the body member so that insertion pointmoves substantially along the body member centerline relative to thebody member; (d) axial rotation of the body about the centerline; and(e) a combination of two or more of motions (a) through (d).
 24. Thesurgical system of claim 20, wherein the drive system is furtheroperatively coupled to both the insertion section and the controlstation so that an operator is able use the input device to operate theinsertion section in such a manner as to rotate the end effectorgenerally axially relative to the distal longitudinal centerline of thebody member within the patient's body distal of the body wall.
 25. Thesurgical system of claim 20, wherein the drive system includes at leastone elongate steering control element couplable to the bendable member,the control element actuatable to steer the bendable member in the aleast one degree of freedom of relative to the body member.
 26. Thesurgical system of claim 25, wherein the at least one elongate steeringcontrol element coupled comprises a plurality of cables.
 27. Thesurgical system of claim 20, wherein the body member includes alongitudinal channel, the bendable member and end effector beingremovably insertable through the longitudinal channel so that the endeffector is extendable distally from the forearm assembly to a pointadjacent the worksite.
 28. The surgical system of claim 20, wherein thedistal longitudinal centerline of the body member is generally alignedcollinearly to the proximal longitudinal centerline of the body member,the body member defining a generally straight body.
 29. The surgicalsystem of claim 20, wherein the insertion section comprises one of asurgical treatment instrument, a diagnostic probe, an endoscope, and acombination of one or more of these.